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ABSTRACT 

In problem-based learning, an instrument is needed to 
measure students' actual learning activities. Mapping the domain of 
learning activities undertaken by students and improving problems is 
especially important in a problem-based curriculum because students 
choose their own learning objectives. Such an instrument, the Topic 
Evaluation Questionnaire (TEQ) , was developed. The TEQ* s reliability 
and validity were assessed, as well as its usefulness as an indicator 
of problem areas in a course's content. Subjects were 125 second-year 
students attending the Medical School of the University of Limburg 
(Netherlands) enrolled in a childbirth and child development course. 
The TEQ contains 134 topics from the course and eight topics 
indirectly related but included to estimate response set effects. For 
each topic, a.Likert-type question was formulated to measure student 
perceptions of time they had spent studying the topic. Students were 
also administered a 222-item true/false; type achievement test on the 
course material. In all, 210 achievement test items were classified 
by a teacher and an independent rater. The TEQ appeared to be 
reliable and was fairly valid. It appeared to be useful in measuring 
problem areas in course content, as well as student's actual learning 
activities. In stvu, the T.E.Q. informs as to whether the goals set by 
teachers are reached by scudents. One table and one bar graph are 
provided. (SLD) 
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Over the past ten years problem-based learning has gained the 
reputation of a far-reaching innovative approach to education (Barrows 
& Tamblyn, 1980). The principal idea behind problem-based learning 
is that learning should be organized around problems which are related 
to the profession, rather than around subjects which are derived from 
academic disciplines. These problems usually consist of a description 
of a set of observable phenomena or events which are in need of some 
kind of explanation. Students analyze these problems, attempting to 
understand the underlying principles or processes through small-group 
discussion. In doing so, they activate whatever they already know 
about the problems. However, students' prior knowledge in itself is not 
sufficient to attain a deep understanding. During discussion usually 
some questions remain unanswered which subsequently serve as a guide 
for independent and self-directed learning (Schmidt, 1982). 

New approaches to professional education no^d careful evaluation. 
This is particularly true for the approach described in this paper. In 
problem-based learning students are responsible for their own learning 
activities because they fonnulate their own le^uning goals. This implies 
that students choose their own topics and also decide how much time 
they will spend on studying each topic. As described above, problems 
are the staiting-point for students' learning activities serving as a guide 
for formulating learning goals. Therefore, a high quality of problems is 



essential for students' learning. Ineffective problems could probably 
lead to selecting learning goals not intended by the teachers. 
Consequently, the self-directed learning approach may cause 
uncertainties about students' competencies among staff and other parties 
concerned. In the long run, students* lack of sufficient knowledge will 
have negative implications for the quality of health care. Therefore, it 
is obvious that problem-based programs need adequate evaluation 
instruments to assess the effectiveness and quality of their innovation. 
These instruments should provide information about students' actual 
learning activities in relation to the problems in order to improve 
ineffective problems. 

Usually, two methods are used to assess the quality of Instruction: 
Educational outcomes, expressed as students' performances on various 
tests, and student ratings of die quality of instruction. Each of these 
methods has its own shortcomings. Achievement tests are usually 
applied to make decisions about students and to distinguish between 
good and poor students. However, achievement test scores by itself can 
not be used to improve curricula in order to enable studems to achieve 
better. Student ratings of the quality of instruction are generally seen 
as an instrument which measures teaching behavior facilitating student 
learning and which in turn may provide data for educational 
improvement (Marsh, 1984). Nevertheless, this method still suffers 
from the limitation that it provides rather global information which does 



not contain detailed information about the nature of the weaknesses. 
Consequently, these instruments can only be used to identify what 
should be improved in a course but not how this should be improved 
(Schmidt, Des Marchais & Black, 1989). This implies that more 
specific information is required. Such information should yield data 
about students* actual learning activities in relation to the contents of 
the course. However, students' actual learning activities, an important 
variable of the educational process, are usually ignored in evaluation 
studies. In general, educational researchers are inclined to neglect 
course content as a variable. 

In summary, an instrument is needed to measure students* actual 
learning activities. Data gathered by means of this instrument should 
be applicable for mapping the domain of learning activities undertaken 
by the students and should be useful to improve problems. Such 
information is especially important in a problem-based learning context 
because of students' responsibility for choosing their own learning 
objectives. In this paper such an instrument is presented. Successively, 
the reliability and validity of this instrument, the Topic Evaluation 
Questionnaire (T.E.Q.), are assessed. The central focus of this paper is 
the usefulness of the T.E.Q. to acquire information about Students* 
learning activities; in other words, is the T.E.Q. an appropriate 
instrument to indicate problem areas in a course content. 



s 

Method 

Subjects 

The study was conducted at the Medical School of the University 
of Limburg, the Netherlands. The first four ye;>rs of the problem- 
based curriculum are structured as a series of six-weeks cours».s. In 
this study, 125 second year students in the 1989-90 academic year 
participated. These students attended a slx-v^eeks course on the nonnal 
pregnancy, delivery and normal development of children and 
adolescents. The course contained thirteen problems concerning topics 
such as aspects of childbirth and psychosexual development. Students 
were randomly assigned to small-group tutorials. 

Instruments 

The instruments used in this study consisted of the T.E.Q. and an 
end-of-unit examination. The T.E.Q. contained a list of 134 topics 
covering the course content as intended by the teachers. In addition, 
eight topics, not directly related to the course content were included to 
estimate response set effects. For each topic a Likert-rype question was 
formulated. Students were asked to indicate whether they had spent: 
No time at all (1), little time (2), a reasonable amount of time (3), 
much time (4) or very much time (5) on studying this particular topic. 
This question intended to measure students' perceptions about the time 
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they had spent on studying each topic related to the course. Time-on- 
task is generally considered as a major determinant of achievement 
(Carrol, 1963). Furthermore, self assessments appeared to be good 
predictors of academic performance (Shrauger ? : Osberg, 1981), 

The other instrument used was an achievement test, including 222 
items of the true-false type. Students' scores n n this achievement test 
consisted of the percentage items correctly answered. 

Proced ure 

At the end of the unit, students were required to fill in the T.E.Q. 
Next, the achievement test was administered to the students. In order to 
avoid that filling in the T.E.Q. could affect students' learning activities, 
the time span between the two measurement conditions was only 24 
hours. 

Analysis 

One of the teacher? and an independent rater were asked to assign 
the items of the test to one of the thirteen problems presented to 
students. Of the total of the 222 items initially included in the 
achievement test, six items were removed because of shon comings in 
their formulation and six items we.e left out because they were related 
to extra-curricular activities. Thus, 210 items were actually classified. 
The two raters agreed about 135 items which could be assigned 



uniquely to one problem. The interrater agreement was equal to .64. 
The topics of the T.E.Q. were initially derived from the thirteen 
problems. Therefore, there was no need to use raters to classify them. 
The analyses in this study were all conducted at the level of singular 
problems. For each problem average topic scores and average 
achievement scores were computed. The reliability of the T.E.Q. and 
the achievement test was determined by estimating coefficients alpha. 

Results and discussion 

Reliability 

The coefficient alpha for the T.E.Q. and the achievement test were 
respectively .98 (134 topics) and .76 (135 items). The average 
coeFcient alpha for each of the problem item sets of the T.E.Q. varied 
between .67 and .91 and for the achievement test between .06 and .58. 
These results supported the reliability of the T.E.Q. and indicated that 
the achievement test's reliability for certain problems was fairly lew. 

Criterion validity 

Time to learn is an important variable to predict educational 
achievement. This means that a relationship between time spent on 
studying topics and the test item scores may be expected. A positive 
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relationship between the T.E.Q. and the performances on the test 
demonstrates the criterion validity of the T.E.Q. 

To show this relationship an ANOVA was conducted. The 
dependent variable consisted 01 the average achievement scores for each 



Table 1: ANOVA-analysis 




of she problems. The independent variables involved (a) the average 
time spent on studying each problem, containing five levels; and (b) the 
thirteen problems. Table 1 shows the results of the ANOVA-analysis. 
These results indeed supported the criterion validity of the T.E.Q. 
Besides, it is interesting to note that the independent variable 
'problems' explained more variance than the independent variable 'time 
spent'. This effect is most probably due to the questionable quality of 
the criterion instrument itself; particularly in view of the low average 



coefficient alpha for some of the problem item sets of the achievement 
test. Another plausible explanation is the number of test items referring 
to the thirteen problems varying from 2 to 22. This seems to illustrate 
that the achievement test did not equally reflect the problems related to 
the course. Consequently, the overlap of the test content with the 
objectives domain specified by the teachers may be very low. 

Discriminatory power 

To judge whether the T.E.Q. is a useful instrument for indicating 
problem areas in a course content, time spent on studying topics 
connected to a problem should differ across problems, The material in 
this study indicated that differences between time spent on certain 
topics indeed differed across problems (F( 12, 1623)= 23.660, p < .000). 
The average time spent varied from 1.94 (50= .65) to 2.96 (5D=.71), 
as indicated in Figure 1. The average time spent on studying the eight 
topics, not intended by the teachers was 1.51 (SD= .59). These results 
demonstrated that the T.E.Q. can be used to indicate problem areas. 

Utility 

The utility of the topic evaluation instrument to improve curricula 
can be illustrated by Figure 1. The results in Figure 1 showed that 
time spent on studying the different tasks varied between "no time at 
all" (1) and "a reasonable amount of time" (3). A reasonable amount 
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Figure 1: Average time spent on studying problems 

of time had been spent on studying task four, seven and el^ht. On the 

contrary, little time had been spent on task rive and task thirteen. 

Consequently, problem five and thirteen need to be improved in order 

to reach the goals set by the teachers. By examining the time students 

had spent on studying each separate topic related to that particular 

problem, teachers can collect information about how these problems 

should be improved. IT s, the T.E.Q. seems to be applicable to 

measure whether students' actual learning activities cover the course 

content. 
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In problem-based learning students are responsible to choose their 
own learning activities. Problems serve as a guide in formulating 
learning goals. This self-directed learning approach may cause a gap 
between students' actual learning activities and those intended by the 
teachers. This implies that an instrument is needed containing 
information about which topics were studied by the students and how 
much time had been spent on studying each topic. This instrument's 
aim is to indicate how problems can be improved in order to reach the 
goals set by the teachers. 

The T.E.Q. described in this study seems to meet these 
requirements. This instrument appeared to be reliable. Furthermore, it 
was shown that the instrument is fairly valid, which prcbably can be 
attributed to the low reliability of some of the problem item sets of the 
achievement test. The T.E.Q. appeared to be useful to indicate problem 
areas and can serve as an instrument to obtain specific information 
about students' actual learning activities. This information is 
particularly needed in an innovative curriculum based on the problem- 
based learning ppproach, because of the self-directed learning method. 
In summary, the T.E.Q. can indicate whether the educational goals set 
by the teachers are really reached by the students. 
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